Conference system, monitoring system, image processing apparatus, image processing method and a non-transitory computer-readable storage medium

ABSTRACT

To provide a conference system, a monitoring system, an image processing apparatus, an image processing method and A non-transitory computer-readable storage medium that stores a computer-image processing program capable of accurately and effectively recognizing an object based on a fisheye-distorted image photographed at a wide angle. 
     When an instruction of selecting an arbitrary point Ci(ui, vi) of a 2D-viewable planar regular image generated based on a fisheye-distorted image S photographed by a fisheye lens is received, point Si(xi, yi) on the fisheye-distorted image S is calculated as a coordinate corresponding to Ci(ui, vi), and a pixel information group configuring the planar regular image is newly generated based on a pixel information group configuring the fisheye-distorted image S.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and animage processing method for processing input image data.

2. Description of the Related Art

A wide-angle lens (such as fisheye lens) or omnidirectional mirror canphotograph a subject at a wide angle, and thus is applied to a cameradevice (such as fisheye lens camera) in a monitoring camera system or TVconference system.

A distortion occurs in a fisheye-distorted image photographed by thefisheye lens camera and a significant distortion occurs particularly inan outer edge (end face) of the image. In order to enhance adiscrimination between subjects in the fisheye-distorted image, there isperformed a fisheye correction processing for converting afisheye-distorted image into a 2D-viewable planar regular image andcorrecting the distortion.

As a technique employing the fisheye correction processing, JapanesePatent Application Laid-Open No. 2008-301034 Publication disclosestherein a technique for outputting and displaying a size-reduced imageof a fisheye lens's total scene and a cut-out distortion-corrected areaof a desired area in the fisheye-distorted image designated by apointing device on the same screen.

SUMMARY OF THE INVENTION

With the technique disclosed in Japanese Patent Application Laid-OpenNo. 2008-301034, however, a specific object displayed in afisheye-distorted image needs to be specified by human eyes and aninstruction needs to be given to a device for monitoring the object, andthus the object cannot be accurately and effectively recognized.

Since a distortion occurs in the fisheye-distorted image as describedabove, when an object recognizing technique for recognizing a specificobject displayed in an image is applied, an object displayed on afisheye-distorted image is difficult to accurately recognize.

The present invention has been made relevant to the above problems, andan object thereof is to provide a conference system, a monitoringsystem, an image processing apparatus, an image processing method and animage processing program capable of accurately and effectivelyrecognizing an object based on a fisheye-distorted image photographed ata wide angle.

According to the present invention, when an arbitrary point on a2D-viewable planar regular image, which is generated based on adistorted circular image photographed by a wide-angle lens oromnidirectional mirror, is instructed to select, pixel information onthe distorted circular image corresponding to pixel information on theplanar regular image indicated by the selection instruction is specifiedand a pixel information group configuring the planar regular image isnewly generated based on a pixel information group configuring thedistorted circular image around the specified pixel information, therebyaccurately and effectively recognizing an object based on afisheye-distorted image photographed at a wide angle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram showing a concept of a basic principle;

FIG. 2 is conceptual diagrams showing an exemplary conversion in which apart or all of a fisheye-distorted image S is cut out and converted intoa planar regular image T;

FIG. 3 is conceptual diagrams showing a conversion into a curved planarregular image;

FIG. 4 is conceptual diagrams showing an exemplary conversion in which apart or entire fisheye-distorted image S is cut out and converted intothe planar regular image T by using the curved coordinate system;

FIG. 5 is conceptual diagrams showing a relationship between a directionin which a fisheye lens camera is arranged and a correction direction;

FIG. 6 is a block diagram showing a structure and functional outline ofan image processing apparatus according to an embodiment;

FIG. 7 is conceptual diagrams showing an example in which a block noiseoccurs in a detailed image of the planar regular image T;

FIG. 8 is a conceptual diagram for explaining a detailed reconversion;

FIG. 9 is a conceptual diagram for explaining the detailed reconversion;

FIG. 10 is a conceptual diagram for explaining the detailedreconversion;

FIG. 11 is diagrams showing exemplary display screens when the planarregular image converted by the basic principle is reconverted;

FIG. 12 is diagrams showing exemplary display screens when the planarregular image converted by the conversion into a curved planar regularimage is reconverted;

FIG. 13 is a flowchart showing operations of the image processingapparatus SS;

FIG. 14 is diagrams showing exemplary screens in which planar regularimages reconverted each time an arbitrary point Ci(ui, vi) is designatedare displayed side by side;

FIG. 15 is diagrams showing exemplary displays in which the planarregular images are reconverted by an object recognizing technique;

FIG. 16 is diagrams showing how a plurality of arbitrary points Ci(ui,vi) are selected by a pointing device (that is, designated by a humanoperation);

FIG. 17 is conceptual diagrams showing an example in which the imageprocessing apparatus SS is applied to a monitoring camera system;

FIG. 18 is conceptual diagrams showing an example in which the imageprocessing apparatus SS is applied to a TV conference system when thefisheye lens camera is arranged sideways;

FIG. 19 is conceptual diagrams showing an example in which the imageprocessing apparatus SS is applied to the TV conference system when thefisheye lens camera is faced upward;

FIG. 20 is a block diagram showing a case in which the image processingapparatus SS is applied to a pan/tilt/zoom camera;

FIG. 21 is conceptual diagrams showing a case in which the imageprocessing apparatus SS is applied to the pan/tilt/zoom camera; and

FIG. 22 is a flowchart showing operations of the image processingapparatus SS when the image processing apparatus SS is applied to thepan/tilt/zoom camera.

DETAILED DESCRIPTION OF THE EMBODIMENT

An embodiment of the present invention will be described below in detailwith reference to the appended drawings. The embodiment described lateris such that the present invention is applied to an image processingapparatus.

[1. Principle of Conversion of Fisheye-Distorted Image]

A) Conversion into Planar Regular Image The image processing apparatusaccording to the present embodiment has, detailed later, a function ofcutting out a part or all of a fisheye-distorted image as exemplarydistorted circular image photographed by a wide-angle lens oromnidirectional mirror and converting it into a 2D-viewable planarregular image.

A basic principle of cutting out a part or all of a fisheye-distortedimage and converting it into a planar regular image (which will besimply called “basic principle”) will be first described with referenceto FIGS. 1 and 2.

FIG. 1 is a conceptual diagram showing a concept of the basic principle.

The basic principle is well known and thus a detailed explanationthereof will be omitted, but in order to obtain a pixel informationgroup configuring a planar regular image, a corresponding coordinate atwhich each item of pixel information is positioned is calculated in afisheye-distorted image corresponding to each item of pixel informationconfiguring the planar regular image.

Specifically, a plane contacting at an arbitrary point on a virtualspherical surface obtained by modeling an optical property of a fisheyelens is defined as a planar regular image, each coordinate on the planeis converted into each coordinate in the fisheye-distorted image by awell-known coordinate conversion and thus the corresponding coordinateis calculated.

In order to cut out apart of the fisheye-distorted image S about thecut-out center point P(xo, yo) and convert it into the planar regularimage T, there is applied a method for calculating a correspondingcoordinate using a virtual spherical surface model defined in the XYZcoordinate system with O shown in FIG. 1 as the origin. In FIG. 2, acoordinate system in which the fisheye-distorted image S is present isassumed as the XY coordinate system and a coordinate system in which theplanar regular image T is present is assumed as the UV coordinatesystem. The XY coordinate system or the UV coordinate system willindicate a coordinate system on the fisheye-distorted image S or theplanar regular image T, respectively, in the following.

With the method for calculating a corresponding coordinate by thevirtual spherical surface model, there is known that assuming that anintersection among P, a straight line parallel to the Z axis and avirtual spherical surface H is Q(xo, yo, zo), the origin G(Xg, Yg, Zg)of the planar regular image T as the corresponding coordinatecorresponding to the cut-out center point P(xo, yo) defined on thefisheye-distorted image S is present on a vector n passing through Q andthe origin in the XYZ coordinate system.

A radius of the fisheye-distorted image S is defined as R, an angleformed between a straight line connecting the cut-out center point P(xo,yo) and the origin OO of the XY coordinate system and the Y axis asazimuth angle α, an angle formed between a straight line connecting theorigin G(Xg, Yg, Zg) as the origin of the UV coordinate system and theorigin O of the XY coordinate system and the Z axis as zenith angle β,and an angle formed between the U axis and an axis passing through the Jaxis (G(xo, yo, zo)), parallel to the XY plane and perpendicular to thestraight line OG (which is also called rotation reference axis) asplanar tilt angle Φ, respectively. Assuming vector U in the U axisdirection in the UV coordinate system and vector J in the rotationreference axis J, the planar tilt angle Φ is defined as an angle formedbetween the vector U and the vector J, and is typically called “planartilt angle.” Thus, it can be seen that the position and the orientationof the planar regular image T in the UV coordinate system are decided bythe three angles including the azimuth angle α, the zenith angle β andthe planar tilt angle Φ. The three angles may be typically called Eulerangle. A magnification m is defined as a parameter indicating acorrection size of the planar regular image T. With a value of themagnification m, the UV coordinate system is arranged such that adistance between OG is m times as long as the radius R. The Euler anglesand the magnification m are variables capable of changing depending onan orientation or size of the planar regular image to be converted. Inother words, the Euler angles and the magnification m function asparameters, and are decided by user's input or device setting.

It is generally known that with the above relationship, point Si(xi, yi)on the XY coordinate system and point Ti(ui, vi) on the UV coordinatesystem are calculated (that is, the corresponding coordinates arecalculated) by Expression (1) and (2) by using parameters A to F and w(Expression (3) to (9)).

x, y, u and v in Expression (1) and (2) correspond to xi, yi, ui and vi,respectively.

$\begin{matrix}{x = {\frac{R\left( {{u\; A} + {v\; B} + {w\; C}} \right)}{\sqrt{u^{2} + v^{2} + w^{3}}} = {G\left( {{u\; A} + {v\; B} + {w\; C}} \right)}}} & {{Expression}\mspace{14mu} (1)} \\{{y = {\frac{R\left( {{u\; D} + {v\; E} + {w\; F}} \right)}{\sqrt{u^{2} + v^{2} + w^{3}}} = {G\left( {{u\; D} + {v\; E} + {w\; F}} \right)}}}{G = \frac{R}{\sqrt{u^{2} + v^{2} + w^{3}}}}} & {{Expression}\mspace{14mu} (2)} \\{A = {{\cos \; \varphi \; \cos \; \alpha} - {\sin \; \varphi \; \sin \; \alpha \; \cos \; \beta}}} & {{Expression}\mspace{14mu} (3)} \\{B = {{{- \sin}\; \varphi \; \cos \; \alpha} - {\cos \; \varphi \; \sin \; \alpha \; \cos \; \beta}}} & {{Expression}\mspace{14mu} (4)} \\{C = {\sin \; {\beta sin\alpha}}} & {{Expression}\mspace{14mu} (5)} \\{D = {{\cos \; {\varphi sin}\; \alpha} + {\sin \; \varphi \; \cos \; \alpha \; \cos \; \beta}}} & {{Expression}\mspace{14mu} (6)} \\{E = {{{- \sin}\; {\varphi sin}\; \alpha} + {\cos \; \varphi \; \cos \; \alpha \; \cos \; \beta}}} & {{Expression}\mspace{14mu} (7)} \\{F = {{- \sin}\; {\beta cos\alpha}}} & {{Expression}\mspace{14mu} (8)} \\{w = {m\; R}} & {{Expression}\mspace{14mu} (9)}\end{matrix}$

There will be described below with reference to FIG. 2 an exemplaryconversion in which the corresponding coordinates are calculated and apart or entire fisheye-distorted image S is cut out and converted intothe planar regular image T.

FIG. 2 is conceptual diagrams showing the exemplary conversion in whicha part or entire fisheye-distorted image S is cut out and converted intothe planar regular image T.

FIG. 2(A) shows the fisheye-distorted image S, and FIGS. 2(B) and 2(C)show images obtained by cutting out a part of the fisheye-distortedimage S and converting it into the planar regular image T.

A size of the image converted from the fisheye-distorted image S intothe planar regular image T (the total number of coordinates on the UVcoordinate system) can be arbitrarily set. That is, an arbitrary rangein the images configuring the fisheye-distorted image S can be convertedinto the planar regular image T.

FIG. 2(B) shows an example in which only Mr. D is selected from amongthe images displayed in the fisheye-distorted image S and is convertedinto the planar regular image T. FIG. 2(C) shows an example in which Mr.B to Mr. E are selected from among the images displayed in thefisheye-distorted image S and are converted into the planar regularimage T.

B) Conversion into Curved Planar Regular Image

As described above, a distortion occurs in a fisheye-distorted image. Itis known that the distortion is larger according to the distance fromthe center of the fisheye-distorted image (that is, toward the outeredge of the fisheye-distorted image).

With the example of FIG. 2, a smaller distortion occurs in FIG. 2(B) inwhich the image around the center of the fisheye-distorted image S isconverted into the planar regular image T and thus a person is shown ina natural form. On the other hand, in FIG. 2(C) in which the images fromthe center of the fisheye-distorted image S to the outer edge areconverted into the planar regular image T, a large distortion is presentin the planar regular image T (that is, Mr. B and Mr. E) correspondingto the images displayed at the outer edge of the fisheye-distorted imageS and the persons are shown to be enlarged in the horizontal directionrelative to the screen. In other words, the persons are shown in anunnatural form.

When an object recognizing technique for recognizing a specific objectdisplayed on an image is applied to the planar regular image having alarge distortion, the objects displayed on the fisheye-distorted imageare difficult to accurately recognize.

In order to specifically explain the images shown in FIGS. 2(B) and2(C), there will be assumed a case in which a pattern matching methodgenerally known as the object recognizing technique (a technique forrecognizing an object by comparing a previously-stored image with animage displayed on a screen in similarity) is applied.

The person image shown in FIG. 2(B) can be generally grasped as person,and is likely to be similar to a previously-stored person image. Thus, aspecific object displayed on an image can be recognized by applying theobject recognizing technique.

On the other hand, the person images shown in FIG. 2(C) are generallydifficult to grasp as persons, and is less likely to be similar to thepreviously-stored person image. Thus, a specific object displayed on animage is difficult to recognize by applying the object recognizingtechnique.

For the above basic principle, a small distortion occurs in the planarregular image as distortion-corrected image in a narrow range (forexample, when the image around the center of the fisheye-distorted imageS is converted into the planar regular image) and thus the basicprinciple can be used with the object recognizing technique, while alarger distortion occurs in the planar regular image asdistortion-corrected image in a wide range (for example, when the imagefrom the center of the fisheye-distorted image S to the outer edge isconverted into the planar regular image) with distance from the centerof the UV coordinate system and thus the basic principle is difficult touse with the object recognizing technique.

In order to improve distortions occurring near the right and leftcontours on the planar regular image T, the present inventors haveinvented a technique for finding the planar regular image T by decidingpixel information at a coordinate C(u, v) in converted pixel informationbased on pixel information near a coordinate P(x, y) obtained by using acorrespondence equation indicating a correspondence between a coordinateC′(u′, v) on the UV curved coordinate system and the coordinate P(x, y)on the XY coordinate system and displaying the image on a displayingpart on the plane based on converted pixel arrangement data (fordetails, see Japanese Patent Application Laid-Open No. 2010-62790Publication, for example).

The converting method disclosed in Japanese Patent Application Laid-OpenNo. 2010-62790 Publication will be simply referred to as “conversioninto a curved planar regular image” below.

The conversion into a curved planar regular image is a well-knowntechnique and thus a detailed explanation thereof will be omitted, butan outline thereof will be described with reference to FIGS. 3 and 4.The same reference numerals as those previously described in FIGS. 3 and4 denote at least the same functions (including operations and effects)and thus a detailed explanation thereof will be omitted.

FIG. 3 is conceptual diagrams showing a conversion into a curved planarregular image.

The conversion into a curved planar regular image defines a virtualspherical surface H having a radius R on the fisheye-distorted image Son the XY plane, Euler angles and a magnification m as shown in FIGS.3(A) and 3(B) (similar to FIG. 2). Similar to FIG. 2, an eye vector npassing through point Q immediately on point P is found, and the UVcoordinate system having an orientation depending on the angle Φ isdefined on the plane T perpendicular to the eye vector n at point Gwhere the OG distance is m·R. The UV coordinate system is curved alongthe side face C of “a virtual column whose point G is one point on theside face and whose center axis is a straight line V′ parallel to the Vaxis and passing through point O”, thereby defining the UV curvedcoordinate system.

With the definition, the UV coordinate system T, and the UV curvedcoordinate C obtained by curving the UV coordinate system T are in apositional relationship shown in FIG. 3(B).

In other words, the UV plane as the coordinate plane of the UVcoordinate system is parallel to the XY plane and an intervaltherebetween is set at w=mR. As illustrated, the U axis is defined inthe horizontal direction along the UV plane, and the V axis is definedin the vertical direction relative to the sheet at point G. The planarregular image T is defined on the UV plane.

On the other hand, the UV curve as the coordinate plane of the UV curvedcoordinate system is obtained by curving the UV plane along the sideface of the virtual column. The virtual column is a column whose centeraxis is the V′ axis (parallel to the V axis and passing through theorigin O) and which has a radius w=mR, and contacts the UV plane at theaxis V. The UV curve coincides with the side face of the virtual column,where the U axis is defined along the arc of the circle having theradius w=mR and the V axis is defined in the vertical direction relativeto the sheet at point G. The curved regular image C is defined on the UVcurve.

In this way, the UV coordinate system and the UV curved coordinatesystem have the common V axis as one coordinate axis and are differentfrom each other in a spatial position of the U axis as the othercoordinate axis. A scale span of the coordinate scale is the same inboth coordinate systems, and an arbitrary point position on thecoordinate plane is indicated with the coordinate (u, v) in bothcoordinate systems. A relationship between two points indicated with thesame coordinate (u, v) on both coordinate systems can be assumed asfollows.

In other words, as shown at the upper left of FIG. 3(B), in terms ofpoint T(u, v) on the UV coordinate system, it can be seen that the pointT(u, v) is on the planar regular image T, is spaced by the coordinatevalue u along the U axis from the origin G and is spaced by thecoordinate value v in the V axis direction.

Then, in terms of point C (u, v) on the UV curved coordinate system, thepoint C(u, v) is on the curved regular image C, is spaced by thecoordinate value u along the arc A from the origin G, and is spaced bythe coordinate value v in the V axis direction.

As described above, since both the point T(u, v) and the point C(u, v)are expressed with the coordinate (u, v) but are defined in differentcoordinate systems, both points are different in spatial positionindicated in the XYZ 3D Cartesian coordinate system. In other words,assuming the coordinate value of the XYZ 3D Cartesian coordinate system,the coordinate of the point T(u, v) corresponds to T(u, v, w) while thecoordinate of the point C(u, v) corresponds to C(u′, v, w′). It can beseen from FIG. 3(B) that u′=w·sin θ and w′=w·cos θ are obtained.

Since the length of the arc GC is equal to the absolute value of thecoordinate value u and the radius of the arc A is w, the angle θ shownin FIG. 3(B) is θ=u/w radians. Thus, it can be seen that u′ and w′ areexpressed as u′=w·sin(u/w) and w′=w·cos(u/w), respectively. Since boththe UV coordinate systems have the V axis as the common coordinate axis,the Y coordinate values of the points T and C take the same coordinatevalue v.

With the above relationship, point Si(xi, yi) on the XY coordinatesystem and point C′(u′, v) on the UV curved coordinate system can befound by the computing Expression (10) and (13) by using parameters A toF and w′ (Expression (3) to (9)). The image near point P is obtained onthe UV curved coordinate system and is developed on the plane T toobtain the planar regular image T.

With the structure, the present inventors have demonstrated thatdistortions are largely improved near the right-side contour or near theleft-side contour of the planar regular image T.

Expression (10) to (13) will be described below. x, y in Expression (10)to (13) correspond to xi, yi described above, respectively.

$\begin{matrix}{u^{\prime} = {{w \cdot \sin}\frac{u}{w}}} & {{Expression}\mspace{14mu} (10)} \\{w^{\prime} = {{w \cdot \cos}\frac{u}{w}}} & {{Expression}\mspace{14mu} (11)} \\{x = {\frac{R\left( {{{u\;}^{\prime}A} + {v\; B} + {{w\;}^{\prime}C}} \right)}{\sqrt{u^{\prime 2} + v^{2} + w^{\prime 2}}} = {G\left( {{{u\;}^{\prime}A} + {v\; B} + {{w\;}^{\prime}C}} \right)}}} & {{Expression}\mspace{14mu} (12)} \\{{y = {\frac{R\left( {{{u\;}^{\prime}D} + {v\; E} + {{w\;}^{\prime}F}} \right)}{\sqrt{u^{\;^{\prime}2} + v^{2} + w^{\prime 2}}} = {G\left( {{u^{\prime}\; D} + {v\; E} + {w^{\prime}F}} \right)}}}{G = \frac{R}{\sqrt{u^{\prime 2} + v^{2} + w^{\prime 2}}}}} & {{Expression}\mspace{14mu} (13)}\end{matrix}$

There will be described below with reference to FIG. 4 an exemplaryconversion in which a part or entire fisheye-distorted image S is cutout and converted into the planar regular image T by using the abovecurved coordinate system.

FIG. 4 is conceptual diagrams showing the exemplary conversion in whicha part or all of the fisheye-distorted image S is cut out and convertedinto the planar regular image T by using the above curved coordinatesystem.

FIG. 4(A) shows the fisheye-distorted image S and FIG. 4(B) shows animage which is obtained by cutting out a part of the image S andconverting it into the planar regular image by using the curvedcoordinate system. As described above, the size of the image when beingconverted from the fisheye-distorted image S into the planar regularimage can be arbitrarily set.

FIG. 4(B) shows an example in which Mr. B to Mr. E are selected fromamong the images in the fisheye-distorted image S and are converted intothe planar regular image. The selected range is the same as the exampleof FIG. 2(C), but it can be seen that little distortion occurs near theright and left contours on the planar regular image shown in FIG. 4(B)as compared with the planar regular image shown in FIG. 2(C).

Any converting methods including the “basic principle” and the“conversion into a curved planar regular image” can be applied in theimage processing apparatus according to the present embodiment, but anexample using the “conversion into a curved planar regular image” willbe described below.

The fisheye-distorted image S is photographed by a camera (such asfisheye lens camera) comprising a wide-angle lens or omnidirectionalmirror, and a relationship between a direction in which the fisheye lenscamera is arranged and a correction direction will be described withreference to FIG. 5.

FIG. 5 is conceptual diagrams showing the relationship between thedirections in which the fisheye lens camera is arranged and thecorrection direction.

FIG. 5(A) shows an example of the correction direction when the fisheyelens camera is arranged sideways (for example, opposite to the Z axis).A distortion correction may be made in the arrow direction in FIG. 5(A)in order to obtain a planar regular image having a wide angle of about180 degrees.

FIG. 5(B) shows an example of the correction direction when the fisheyelens camera is faced upward (for example, in parallel to the Z axis). Adistortion correction may be made in the arrow direction in FIG. 5(B) inorder to obtain a planar regular image having a wide angle of about 360degrees.

In other words, when the fisheye lens camera is faced sideways, acorrection is made in the arrow direction as shown in FIG. 5(A), therebygenerating a panoramic planar regular image with about 180 degrees. Whenthe fisheye lens camera is faced upward, a correction is made in thearrow direction of FIG. 5(B), thereby generating a panoramic planarregular image with about 360 degrees.

[2. Structure and Functional Outline of Image Processing Apparatus]

The image processing apparatus according to the present embodiment has afunction of performing a processing of cutting out a part or all of afisheye-distorted image as exemplary distorted circular imagephotographed by the wide-angle lens or omnidirectional mirror andconverting it into a 2D-viewable planar regular image (which will besimply called “conversion” below) and a processing of reconverting thephotographed fisheye-distorted image into a planar regular image aboutan arbitrary point on the planar regular image (which will be simplycalled “reconversion” below).

The structure and functional outline of the image processing apparatushaving the function according to the present embodiment will bedescribed with reference to FIG. 6 and others.

FIG. 6 is a block diagram showing the structure and functional outlineof the image processing apparatus according to the present embodiment.

As shown in FIG. 6, the image processing apparatus SS includes afisheye-distortion correcting unit 1 (an exemplary conversion unit), acorrection parameter calculating unit 2, a parameter inputting unit 3and others.

The fisheye-distortion correcting unit 1 includes a correspondingcoordinate calculating unit 4, a planar regular image creating unit 5, afisheye-distorted image memory 6 (an exemplary distorted circular imagestoring unit), a planar regular image memory 7 (an exemplary planarregular image storing unit) and others for executing a function ofperforming a processing of calculating a corresponding coordinate atwhich each item of pixel information is positioned on thefisheye-distorted image corresponding to each item of pixel informationconfiguring the planar regular image.

Specifically, when surrounding subjects are photographed at 180 degreesby an optical lens (not shown) and the fisheye-distorted image S as anexemplary distorted circular image having the radius S is converted intodigital data by an image sensor (not shorn) in the fisheye-distortioncorrecting unit 1, the converted fisheye-distorted image S is stored inthe fisheye-distorted image memory 6. The fisheye-distortion correctingunit 1 converts the converted image into the planar regular image T as apixel information group capable of being displayed on a display device(not shown) and stores the planar regular image T in the planar regularimage memory 7.

The fisheye-distorted image memory 6 is configured of a typical buffermemory for image data storage, and stores the fisheye-distorted image Sas a pixel information group configured of a set of many pixels arrangedat the coordinates (x, y) on the XY coordinate system. Thefisheye-distorted image S is an image having the radius S about theorigin O in the XY coordinate system as described above.

The planar regular image memory 7 is configured of a typical buffermemory for image data storage, and stores the planar regular image T asa pixel information group configured of a set of many pixels arranged atthe coordinates (u, v) on the UV coordinate system. The contour of theplanar regular image T can be arbitrarily set. The pixel informationstored in the planar regular image memory 7 is used for both the planarregular image T and the curved regular image C described above.

As stated in [1. Principle of conversion of fisheye-distorted image],the UV coordinate system and the UV curved coordinate system are commonin that both are a 2D coordinate system having the U axis and the Vaxis, and the planar regular image T defined on the UV coordinate systemand the curved regular image C defined on the UV curved coordinatesystem are common in that both are configured of an arrangement of manypixels arranged at the coordinate values (u, v). Thus, the planarregular image T can be obtained by displaying an image on a plane basedon the pixel arrangement data stored in the planar regular image memory7 while the curved regular image C can be obtained by displaying animage on a column side face. The pixel arrangement data stored in theplanar regular image memory 7 can be data for the planar regular image Tand data for the curved regular image C.

The corresponding coordinate calculating unit 4 uses the parameters (thevalues of A to F, α, β, Φ in Expression (3) to (8)) set by thecorrection parameter calculating unit 2 detailed later and the presetradius R to calculate the corresponding coordinate (x, y) on thefisheye-distorted image S corresponding to an arbitrary coordinate (u,v) on the planar regular image given by the planar regular imagecreating unit 5 detailed later.

In other words, the corresponding coordinate calculating unit 4 has afunction of, when an arbitrary coordinate (u, v) on the planar regularimage is given, returning the corresponding coordinate (x, y) on thefisheye-distorted image S corresponding to the arbitrary coordinate tothe planar regular image creating unit 5. The arbitrary coordinate (u,v) on the planar regular image is generated by the pixels of the outputplanar regular image by an uv coordinate value generating unit 51described later.

More specifically, the corresponding coordinate calculating unit 4 has afunction of calculating the corresponding coordinate (x, y)corresponding to the coordinate (u, v) by the computations based on therelational Expression (equation (1) to equation (9)) in the basicprinciple or the relational Expression (equation (10) to equation (13))in the conversion into a curved planar regular image, and includes acommon coefficient computing part 41, a curved coordinate correctingpart 42, and an xy coordinate calculating part 43.

When being given the magnification m from a parameter inputting unit 100and the coordinate v from a planar regular image creating unit 300, thecommon coefficient calculating part 41 uses the preset radius R of thedistorted circular image to perform a processing of substituting thevalue calculated in equation (9) into G and computing the commoncoefficient G.

The curved coordinate correcting part 42 has a function of making acalculation necessary for correcting the coordinate (u, v) on the 2D UVCartesian coordinate system defined on a plane into the coordinate (u,v) on the 2D UV curved coordinate system defined on a column side face,and calculates u′ and w′ based on Expression (9), (10) and (11) by usingthe preset radius R of the fisheye-distorted image when being given themagnification m from the parameter inputting unit 100 and the coordinateu from the planar regular image creating unit 300.

When the relational Expression (Expression (1) to (9)) in the basicprinciple are used, the curved coordinate correcting part 42 is omittedand u, v, ware input into the xy coordinate calculating part 43 insteadof u′, v, w′.

The xy coordinate calculating part 43 uses the coordinate v given fromthe planar regular image creating unit 5, A to F (Expression (3) to (8))input from the correction parameter calculating unit 2, the commoncoefficient G calculated by the common coefficient computing part 41, u′and w′ calculated by the curved coordinate correcting part 42, and thepreset radius R of the fisheye-distorted image to calculate and output xand y based on Expression (12) and (13) to the planar regular imagecreating unit 5.

The planar regular image creating unit 5 uses the values output from thexy coordinate calculating part 43 to create and store the planar regularimage into the planar regular image memory 7. More specifically, first,the coordinate (u, v) of the first pixel of interest configuring theplanar regular image is given to the corresponding coordinatecalculating unit 4 to calculate the corresponding coordinate (x, y). Aprocessing of reading a pixel value of a pixel arranged near thecorresponding coordinate (x, y) of the fisheye-distorted image S storedin the fisheye-distorted image memory 6 and deciding a pixel value of apixel of interest based on the read pixel value is performed per pixelconfiguring the planar regular image, and the pixel value of each pixelis stored in the planar regular image memory 7 to create the planarregular image.

In order to realize the functions, the planar regular image creatingunit 5 includes the uv coordinate value generating part 51, a planarregular image memory controller 52 (an exemplary screen informationgeneration unit), a pixel value deciding part 53, and afisheye-distorted image memory controller 54.

The uv coordinate value generating part 51 generates a value of uv(coordinate value) for the pixels configuring the output planar regularimage. For example, when the size of the output planar regular image is400 dots horizontal×300 dots vertical, the values of u=−200 to 200 andv=−150 to 150 are output.

The planar regular image memory controller 52 is a control device forwriting and reading data into and from the planar regular image memory7, and when a pixel value of a specific pixel is decided by the pixelvalue deciding part 53, performs a processing of writing the decidedpixel value in the specific pixel stored in the planar regular imagememory 7. In this way, when the processing of writing the pixel value iscompleted for all the pixels, the planar regular image T is created inthe planar regular image memory 7. The planar regular image memorycontroller 52 reads and outputs data on the planar regular image T to adisplaying part (not shown), and displays the planar regular image T ona display screen.

The fisheye-distorted image memory controller 54 is a control device forwriting and reading data into and from the fisheye-distorted imagememory 6. As described above, the data on the input fisheye-distortedimage S is stored in the fisheye-distorted image memory 6 by thefisheye-distorted image memory controller 54.

The fisheye-distorted image memory controller 54 can read and output thedata on the fisheye-distorted image S stored in the fisheye-distortedimage memory 6 to the display device, and can display thefisheye-distorted image S on the display screen as needed.

Further, when being given a coordinate (x, y) from the correspondingcoordinate calculating unit 4, the fisheye-distorted image memorycontroller 54 also serves to read a pixel value of a pixel positionednear the coordinate (x, y) from the data on the fisheye-distorted imageS stored in the fisheye-distorted image memory 6 and to give the pixelvalue to the pixel value deciding part 53.

There will be described a processing of converting the fisheye-distortedimage S into the planar regular image T by the thus-configured planarregular image creating unit 5.

First, the uv coordinate value generating part 51 generates a coordinate(u, v) indicating a specific pixel of interest on the pixel arrangementconfiguring the planar regular image T. The generated coordinate (u, v)is given from the uv coordinate value generating part 51 to thecorresponding coordinate calculating unit 4. Thereby, a correspondingcoordinate (x, y) corresponding to the coordinate (u, v) is calculatedand the corresponding coordinate (x, y) is given to thefisheye-distorted image memory controller 54.

As stated above, the fisheye-distorted image memory controller 54 readsthe pixel value of the pixel positioned near the coordinate (x, y) fromthe data on the fisheye-distorted image S stored in thefisheye-distorted image memory 6 and gives it to the pixel valuedeciding part 53.

The fisheye-distorted image S is configured of a set of many pixelsarranged at the coordinates (x, y) on the 2D XY Cartesian coordinatesystem, and is actually configured of the digital data defining theunique pixel values at many lattice points arranged in a matrix atpredetermined pitch. Therefore, the corresponding coordinates (x, y)calculated by the corresponding coordinate calculating unit 4 aretypically positioned between multiple lattice points.

Thus, when actually deciding the pixel value of the pixel of interest onthe planar regular image T arranged at the position of the coordinate(u, v), the pixel value deciding part 53 reads the pixel values of thereference pixels on the fisheye-distorted image S arranged near theposition of the corresponding coordinate (x, y), and makes aninterpolating calculation (such as well-known bilinear interpolatingmethod, bicubic/spline interpolating method) on the pixel values of thereference pixels. A pixel value of a pixel nearest to the positionindicated by the corresponding coordinate (x, y) may be decided as thepixel value of the pixel of interest without performing theinterpolation.

The pixel value of the pixel of interest decided by the pixel valuedeciding part 53 in this way is input into the planar regular imagememory controller 52. On the other hand, the generated coordinate (u, v)is input from the uv coordinate value generating part 51 into the planarregular image memory controller 52.

The planar regular image memory controller 52 performs a processing ofwriting the pixel value decided by the pixel value deciding part 53 asthe pixel value of the pixel of interest positioned at the coordinate(u; v) of the planar regular image T stored in the planar regular imagememory 7.

Though a pixel value of a pixel of interest is decided and written asdescribed above, the uv coordinate value generating part 51 sequentiallygenerates the coordinates (u, v) indicating all the pixels on the pixelarrangement configuring the planar regular image T and the pixel valuesof the individual pixels are decided and stored in the planar regularimage memory 7.

The parameter inputting unit 3 includes a keyboard, a mouse, atrackball, a touch panel or an electronic pen, and can designate onepoint on the fisheye-distorted image S displayed on the display deviceor the planar regular image T subjected to the conversion processing inresponse to a user's operation.

More specifically, the three parameters including the eye vector n, theplanar tilt angle Φ and the magnification m are needed for calculatingthe corresponding coordinate by using Expression (1) and (2) or (12),(13) or the like. The parameter inputting unit 3 is a component forinputting the three parameters based on a user's instruction. In otherwords, the parameter inputting unit 3 serves to input the eye vector nfacing in an arbitrary direction with the origin O as starting point onthe 3D XYZ Cartesian coordinate system as the parameter indicating wherethe planar regular image is cut out, to input the predetermined planartilt angle Φ as the parameter indicating an orientation in which theplanar regular image is cut out, and to input the predeterminedmagnification m as the parameter indicating the correction size of theplanar regular image.

In the present embodiment shown herein, the display device can displaythe fisheye-distorted image S stored in the fisheye-distorted imagememory 6 as needed. The parameter inputting unit 3 receives the user'sinput instruction of designating one point on the fisheye-distortedimage S displayed on the display device thereby to grasp the position ofthe point as cut-out center point P(x0, y0), and receives the cutoutcenter point as the parameter indicating the eye vector n thereby tooutput it to the correction parameter calculating unit 2.

The correction parameter calculating unit 2 includes a correctionparameter calculating part 21 (an exemplary selection-instructionreceiving unit) and a corresponding coordinate calculating unit 22 (anexemplary corresponding coordinate specifying unit).

When being given the eye vector n and the planar tilt angle Φ from theparameter inputting unit 3, the correction parameter calculating part 21finds the azimuth angle α and the zenith angle β based on the eye vectorn, and calculates the rotation coefficients A, B, C, D, E, F based onExpression (3) to (8).

As described above, the image processing apparatus according to thepresent embodiment has a function of reconverting the fisheye-distortedimage S stored in the fisheye-distorted image memory 6 into the planarregular image T about an arbitrary point on the planar regular image Tconverted from the fisheye-distorted image S.

The correction parameter calculating part 21, the correspondingcoordinate calculating unit 22 and the fisheye-distortion correctingunit 1 (an exemplary reconverting unit) are provided for realizing thereconversion.

In many cases, for displaying the planar regular image T obtained by theabove conversion on the display device, the planar regular image T hasto be reduced in size to a displayable area of the display device fordisplay. Thus, the displayed planar regular image T is displayed byinterpolating a part of the pixel information for reducing the originalimage in its size.

When the planar regular image T from which a part of the pixelinformation is interpolated is cut out and displayed in a predeterminedrange or is displayed in an enlarged manner (that is, when a detailedimage of the planar regular image T is displayed), a block distortion(block noise) occurs, which remarkably deteriorates a discriminationbetween subjects on an image.

The block distortion is also called block noise, and is a phenomenonthat partial areas in an image seem mosaic-like in a video.

An example in which a block noise occurs in the detailed image of theplanar regular image T will be described herein with reference to FIG.7.

FIG. 7 is conceptual diagrams showing an example in which a block noiseoccurs in the detailed image of the planar regular image T. The crossmarks in the upper diagrams of FIGS. 7(A) and 7(B) indicate an area towhich an instruction is given by the parameter inputting unit 3 forcutting out and displaying a predetermined range (such as an instructionby the pointing device).

The upper diagram of FIG. 7(A) shows the example in which thefisheye-distorted image S is converted into the planar regular image Tby the basic principle and the lower diagram of FIG. 7(A) shows theexample in which a part of the converted planar regular image T is cutout and displayed.

From the example of the lower diagram of FIG. 7(A), it can be seen thata block noise occurs in the subject in the cut-out and displayed planarregular image T and the contour of the subject is largely distorted.

The upper diagram of FIG. 7(B) shows the example in which thefisheye-distorted image S is converted into the planar regular image Tby the conversion into a curved planar regular image, and the lowerdiagram of FIG. 7(B) shows the example in which a part of the convertedplanar regular image T is cut out and displayed.

From the example of the lower diagram of FIG. 7(B), it can be seen thata block noise occurs in the cut-out and displayed planar regular image Tand the contour of the subject is largely distorted.

The fisheye-distorted image S is converted into the planar regular imageT in order to convert a photographed object into the planar regularimage T capable of being easily monitored and to present it to the usersince a significant distortion occurs in the fisheye-distorted image S.In order to keep monitoring it or to obtain more detailed information onthe object photographed in the planar regular image T, it is desirablethat a predetermined range of the object photographed in the planarregular image T is cut out and displayed or is displayed in an enlargedmanner. It is desirable that the object recognizing technique is appliedto the planar regular image T to accurately recognize the object on thefisheye-distorted image without human operation.

However, it is difficult for the user to obtain detailed information onthe image in which the block noise occurs and to apply the objectrecognizing technique.

The image processing apparatus SS according to the present embodiment isconfigured to perform the reconversion by using the correction parametercalculating part 21, the corresponding coordinate calculating unit 22and the fisheye-distortion correcting unit 1.

The reconversion will be described in detail with reference to FIGS. 8to 10.

FIGS. 8 to 10 are conceptual diagrams for explaining the detailedreconversion.

The reconversion may employ both the converting method with the basicprinciple and the converting method with the conversion into a curvedplanar regular image, and the converting method with the conversion intoa curved planar regular image will be described below.

There will be assumed a case in which a fisheye-distorted image S issubjected to the conversion processing, that is, converted into a curvedplanar regular image C by the above processing and the curved planarregular image is displayed on the displaying part. It is assumed hereinthat the curved planar regular image C about the cut-out center P(xo,yo) is displayed as shown in FIG. 8. The cut-out center P(xo, yo) may bedesignated based on the user's operation of the parameter inputting unit3 or may be designated based on the position preset by the imageprocessing apparatus SS.

The user can select an arbitrary point on the curved planar regularimage C based on the operation of the parameter inputting unit 3. By wayof example, the arbitrary position may be a point where the user maywant to obtain detailed information on the object displayed in thecurved planar regular image C, or may be the center of gravity of anobject when the object recognizing technique is applied to recognize thepredetermined object.

In the example of FIG. 8, the planar regular image about point G isconverted. The operation has been previously described, but the center(point P) of the planar regular image is designated on thefisheye-distorted image S, point Q on the virtual spherical surfacecorresponding to point P is decided, and α and β are calculated frompoint Q. Then, the tilt angle Φ of the corrected image is designated andthe coefficients A to F are calculated by Expression (3) to (8). Thecorresponding coordinates of the fisheye-distorted image and the planarregular image are calculated by using A to F and the planar regularimage about point G is obtained.

In the example of FIG. 8, moreover, it is assumed that Ci (ui, vi) isselected as the arbitrary point.

When receiving Ci (ui, vi), the parameter inputting unit 3 outputs theinput information to the correction parameter calculating part 21.Specifically, the parameter inputting unit 3 outputs the threeparameters including the eye vector n, the planar tilt angle Φ and themagnification m to the correction parameter calculating part 21 inresponse to a user's instruction.

When receiving an input of a point (an input of Ci (ui, vi)) on thefisheye-distorted image S designated by the parameter inputting unit 3,the correction parameter calculating part 21 calculates themagnification m, the radius R of the fisheye-distorted image, and A to F(Expression (3) to (8)) from the point designated on thefisheye-distorted image S, and outputs them to the correspondingcoordinate calculating unit 22.

The corresponding coordinate calculating unit 22 uses the inputmagnification m, radius R of the fisheye-distorted image, A to F(Expression (3) to (8)) and Expression (12), (13) to calculate pointSi(xi, yi) on the fisheye-distorted image S as the correspondingcoordinate of Ci(ui, vi).

In other words, the corresponding coordinate calculating unit 22specifies the pixel information on the fisheye-distorted image Scorresponding to the pixel information on the curved planar regularimage C.

Specifically, the corresponding coordinate calculating unit 22 specifiesa point on the fisheye-distorted image S corresponding to the cut-outcenter Ci(ui, vi).

That is, as shown in FIG. 9, point Qi on the virtual spherical surfacecorresponding to point Si(xi, yi) is decided and α′ and β′ arecalculated from point Qi. Then, the tilt angle Φ′ of the corrected imageis designated and the coefficients A to F (Expression (3) to (8)) arenewly calculated. Point Si(xi, yi) on the fisheye-distorted image S iscalculated as the corresponding coordinate of Ci(ui, vi) and theinformation on it is output to the fisheye-distortion correcting unit 1.

The fisheye-distortion correcting unit 1 (an exemplary reconvertingunit) sequentially generates the coordinates (u, v) indicating all thepixels on the pixel arrangement configuring the planar regular image Tabout Ci(ui, vi) as shown in FIG. 10 (that is, like the aboveconversion), and the pixel values are determined for the individualpixels and are stored in the planar regular image memory 7.

In this way, the planar regular image about point Ci(ui, vi) can beobtained by the image processing apparatus SS.

The exemplary display screens displayed during the reconversion will bedescribed with reference to FIGS. 11 and 12.

FIG. 11 is diagrams showing an exemplary display screen when the planarregular image converted by the basic principle is reconverted, and FIG.12 is diagrams showing an exemplary display screen when the planarregular image converted by the conversion into a curved planar regularimage is reconverted.

FIG. 11(A) shows how an arbitrary point is selected in the planarregular image converted by the basic principle. The cross mark in FIG.11(A) indicates an area to which an instruction for the arbitrary pointis given by the parameter inputting unit 3 (such as instruction by thepointing device).

FIG. 11(B) is a conceptual diagram showing how a correspondingcoordinate of the fisheye-distorted image S is calculated in apredetermined range about the arbitrary point. The predetermined rangecan be arbitrarily set by the operation of the UV coordinate valuegenerating part 51. The predetermined range is set to be wide so thatthe layout size of the reconverted planar regular image is enlarged(that is, the number of pixels configuring the planar regular imageincreases).

As a specific operation, a corresponding position in thefisheye-distorted image is calculated from the angles α, β, Φ andExpression (1), (2) when the position designated in FIG. 11(A) issubjected to distortion correction, as described above. The cut-out area(area to be converted) in the fisheye-distorted image about thecalculated position is indicated by a dotted line. The cross mark inFIG. 11(B) indicates a position corresponding to the fisheye-distortedimage at the position of interest in FIG. 11(A). The angles α′, β, Φ′are newly calculated from the position of the cross mark in thefisheye-distorted image S thereby to update (calculate) the coefficientsA to F. The new coefficients A to F and Expression (1), (2) are used tomake the distortion correction again and to generate a cut-out correctedimage.

FIG. 11(C) shows a reconverted planar regular image. The area indicatedby the dotted line in FIG. 11(B) is reconverted to be displayed.

FIG. 12(A) shows how an arbitrary point is selected in a curved planarregular image converted by using the conversion into a curved planarregular image. The cross mark in FIG. 12(A) is similar to that in FIG.11.

FIG. 12(B) is a conceptual diagram showing how a correspondingcoordinate in the fisheye-distorted image S is calculated in apredetermined range about the arbitrary point.

As a specific operation, a corresponding position in thefisheye-distorted image is calculated from the angles α, β, Φ andExpression (12), (13) when the position designated in FIG. 12(A) issubjected to distortion correction, as described above. The cut-out areain the fisheye-distorted image about the calculated position isindicated by a dotted line. The cross mark in FIG. 12(B) indicates aposition corresponding to the fisheye-distorted image at the position ofinterest in FIG. 12(A). The angles α′, β′, Φ′ are newly calculated fromthe position of the cross mark in the fisheye-distorted image S therebyto update (calculate) the coefficients A to F. The new coefficients A toF and Expression (12), (13) are used to make the distortion correctionagain and to generate a cut-out corrected image.

FIG. 12(C) shows the reconverted planar regular image. The areaindicated by the dotted line in FIG. 12(B) is reconverted to bedisplayed.

With the above structure, the image processing apparatus SS according tothe present embodiment can accurately and effectively recognize anobject based on the fisheye-distorted image S photographed at a wideangle. This is because the fisheye-distorted image S is subjected againto the distortion correction to be displayed about the position of thearea of interest in the curved planar regular image C and thus the imageprocessing apparatus SS is not influenced by a deterioration in imagequality due to a downsizing operation when the curved planar regularimage C is displayed to be monitored. When the cut-out corrected imageis subjected to digital zooming, a difference between the cut-outmethods causes a difference in image quality to be conspicuous. Further,the curved planar regular image C is obtained by the distortioncorrection and thus a conventional recognizing technique is applicableso that an area of interest can be automatically designated by therecognition. Thus, an object to be monitored can be automaticallytraced. Further, a person does not need to always monitor the object,thereby avoiding an operational error. Since a plurality of cut-outcorrected images at multiple points can be displayed, multiple objectsto be monitored can be addressed.

In order to realize a part or entire functions of the fisheye-distortioncorrecting unit 1, the correction parameter calculating unit 2 and theparameter inputting unit 3, dedicated hardware designed to perform eachprocessing can be applied to a part or entire fisheye-distortioncorrecting unit 1, the correction parameter calculating unit 2 and theparameter inputting unit 3 (so-called hardware acceleration). This isbecause since the fisheye-distortion correcting unit 1 and other arerealized by the dedicated hardware so that computations can be made inparallel by parallel circuits in the hardware, processing can beperformed at a faster speed than instructions are sequentially executed(for example, software is executed by a CPU). Of course, the controldevices such as CPU, RAM and ROM (not shown) mounted on the imageprocessing apparatus SS can perform the functions of a part or entirefisheye-distortion correcting unit 1, the correction parametercalculating unit 2 and the parameter inputting unit 3 under control ofthe software.

The image processing apparatus according to the present invention hasthe function of converting a part of a fisheye-distorted imagephotographed by a fisheye lens into a planar regular image, but an imageto be converted by the apparatus is not limited to an image photographedby a fisheye lens. For example, any image to which a semisphericalprojecting model like a fisheye lens is applied, such as imagephotographed by a convex mirror can be converted by using the imageprocessing apparatus according to the present invention.

[3. Operations of Image Processing Apparatus SS]

The operations of the image processing apparatus SS will be describedbelow with reference to FIG. 13.

FIG. 13 is a flowchart showing the operations of the image processingapparatus SS.

At first, in order to perform the above conversion for converting thefisheye-distorted image S and displaying the planar regular image T orthe curved planar regular image C (which will be collectively called“planar regular image” below) on the displaying part, when a cut-outcenter P (xo, yo) of the fisheye-distorted image S is selected based onthe user's operation of the parameter inputting unit 3 or by using avalue preset in the image processing apparatus SS (step S1), theselected information is output as the parameter to the correctionparameter calculating part 21.

When being given the eye vector n and the planar tilt angle Φ from theparameter inputting unit 3, the correction parameter calculating part 21calculates the azimuth angle α and the zenith angle β based on the eyevector n and calculates the rotation coefficients A, B, C, D, E, F basedon Expression (3) to (8) (step S2) to output the calculated values tothe fisheye-distortion correcting unit 1.

Then, the fisheye-distortion correcting unit 1 uses the calculationresults (α, β, Φ) input from the correction parameter calculating part21 to convert the fisheye-distorted image S into the planar regularimage, and the planar regular image memory controller 52 displays theplanar regular image on the display screen (step S4).

When the user designates an arbitrary point Ci(ui, vi) of the displayedplanar regular image based on the operation of the parameter inputtingunit 3 (step S5: YES), the corresponding coordinate calculating unit 22decides point Qi on the virtual spherical surface corresponding to pointSi(xi, yi) to calculate α′, β′ from point Qi. Then, the tilt angle Φ′ ofthe corrected image is designated and the coefficients A to F are newlycalculated (by Expression (3) to (8)). Point Si(xi, yi) on thefisheye-distorted image S as a corresponding coordinate of Ci(ui, vi) iscalculated and the information on it is output to the fisheye-distortioncorrecting unit 1 (steps S6 and S7).

In step S8, the fisheye-distortion correcting unit 1 generates a planarregular image about point Ci(ui, vi) based on the information input fromthe corresponding coordinate calculating unit 22 (that is, makes thedistortion correction by using α′, β′, Φ′).

The planar regular image memory controller 52 displays the reconvertedplanar regular image on the display screen (step S9).

[4. Display Form of Reconverted Planar Regular Image]

A display form of the reconverted planar regular image will be describedbelow with reference to FIGS. 14 to 16.

The image processing apparatus according to the present embodiment candesignate arbitrary points Ci(ui, vi) of multiple planar regular imagesto reconvert them, and can display the converted planar regular imageson the displaying part.

Specifically, for example, each time the arbitrary point Ci(ui, vi) isdesignated, point Si(xi, yi) on the fisheye-distorted image Scorresponding to the designated point Ci(ui, vi) is calculated therebyto generate a planar regular image about point Ci(ui, vi) (an exemplaryscreen information of the present application). The generated planarregular images may be displayed side by side on the displaying part.

The planar regular images may be generated by the planar regular imagememory controller 52 like the above conversion.

FIG. 14 is diagrams showing an exemplary screen on which the planarregular images reconverted each time an arbitrary point Ci(ui, vi) isdesignated are displayed side by side. The broken lines in FIG. 14indicate that objects to be reconverted are selected.

FIG. 14(A) shows how Mr. C and Mr. E are sequentially designated as thearbitrary point Ci(ui, vi).

The generated planar regular images are displayed side by side in FIG.14(B). In the example of FIG. 14(B), Mr. F looks larger than Mr. C dueto a distance between the camera and the subjects.

FIG. 14(C) shows an exemplary display when only Mr. F is reduced in sizeto be displayed and Mr. C and Mr. F are matched with each other in size.In this way, the magnification m has only to be changed in order tochange the display size of the subjects.

In order to change the magnification m, the correction parametercalculating part 21 may calculate the magnification m under control ofthe CPU (not shown) or a display control unit may be newly provided inthe block diagram of FIG. 6 to cause the unit to change themagnification m. The change of the magnification m may be performed bythe planar regular image memory controller 52 as an exemplary displaycontrolling unit.

FIG. 14(D) shows an exemplary display when an area in the planar regularimage T to be reconverted is enlarged and only the screen on which Mr. Cis displayed (the left window size) is enlarged (that is, the displayarea is enlarged). In order to enlarge the display area, the coordinates(u, v) for the enlarged display area may be generated by the uvcoordinate value generating part 51. The generated coordinates (u, v)are given from the uv coordinate value generating part 51 to thecorresponding coordinate calculating unit 4 to calculate thecorresponding coordinates (x, y) corresponding to the coordinates (u,v), and consequently the planar regular image is reconverted.

The display area of Mr. C in FIG. 14(D) is enlarged compared with thedisplay area in which Mr. C is displayed in FIG. 14(B). It can be seenthat even the shoulders of Mr. B and Mr. D, not displayed in FIG. 14(B),are displayed in the display area of Mr. C in FIG. 14(D).

FIG. 14(E) shows an example in which the display area and the displaysize of Mr. F are enlarged, respectively, as a result of an increase inthe magnification m and an enlargement in the display area.

By displaying as the above described, a more detailed image of thesubject displayed in the display area can be obtained.

There will be described below with reference to FIG. 15 an example inwhich the subject displayed in the reconverted planar regular image isrecognized by an object recognizing technique to be reconverted anddisplayed with a predetermined position of the recognized subject as thearbitrary point Ci (ui, vi).

FIG. 15 is diagrams showing an exemplary display subjected to thereconversion by the object recognizing technique.

The object recognizing technique is well known and thus a detailedexplanation thereof will be omitted, but an object is first recognizedby a pattern matching method or characteristic point extracting methodgenerally known as the object recognizing technique. A preset point(such as center of gravity of the object) in the recognized object maybe reconverted to be displayed as the arbitrary point Ci (ui, vi). Theobject recognizing structure is such that object recognizing unit formedof the CPU and the like is added to the exemplary structure of FIG. 6,for example.

FIG. 15(A) shows that a rectangular solid line in the planar regularimage indicates an image of a recognized person's face by the objectrecognition. The dotted lines indicate cut-out areas of thecorresponding fisheye-distorted image S.

FIG. 15(B) shows an exemplary display subjected to the reconversion(that is, an exemplary cut-out corrected image display). As shown inFIG. 15(B), a plurality of objects can be addressed. Even when theobject is moving, it can be automatically traced. The automatic tracingcan be realized by calculating an optical flow and performing awell-known trace processing by the characteristic point extractingmethod.

The arbitrary points Ci(ui, vi) of the image to be reconverted may beselected by the pointing device (that is, designated by a humanoperation).

FIG. 16 shows how the arbitrary points Ci(ui, vi) are selected by thepointing device (that is, designated by a human operation).

FIG. 16(A) shows a cross mark (pointing marker) indicating that a dottedrectangular area in the planar regular image is designated as a cut-outarea by the pointing device.

FIG. 16(B) shows an exemplary display subjected to the reconversion (anexemplary cut-out corrected image display).

The cut-out area designated by the pointing device is displayed in a newwindow thereby to address a plurality of objects. Not only the cut-outcorrected image is displayed on a new window but also a cut-outcorrected image whose position of interest is updated can be displayedin a designated window. Even when the object to be monitored is moving,the cut-out area can be manually updated and traced.

[5. Exemplary Application to Monitoring Camera System]

An example in which the image processing apparatus SS is applied to amonitoring camera system will be described below with reference to FIG.17.

FIG. 17 is conceptual diagrams showing an example in which the imageprocessing apparatus SS is applied to a monitoring camera system.

The image processing apparatus SS according to the present embodimentfurther includes a digital camera (any device capable of outputting afisheye-distorted image to the image processing apparatus SS may beemployed) mounting a fisheye lens (a wide-angle lens or omnidirectionalmirror may be employed) thereon and the display device, and additionallyobject recognizing unit for recognizing a specific object displayed inthe converted planar regular image and an operation recognizing unit forrecognizing a successive change indicated by the recognized object inthe planar regular image per planar regular image in addition to thestructure of FIG. 6. Pixel information on the fisheye-distorted imagecorresponding to an arbitrary point on the object per planar regularimage recognized by the operation recognizing unit is specified to bereconverted based on the specified position. The object recognizing unitand the operation recognizing unit may be realized by the CPU and thelike.

FIG. 17(A) shows a fisheye-distorted image and FIG. 17(B) shows theconverted planar regular image (that is, the planar regular imagegenerated from the fisheye-distorted image by the image processingapparatus SS).

As shown in FIG. 17(A), the fisheye-distorted image is largely distortedaccording to distance from its center and thus is difficult to use withthe conventional object recognizing technique, as stated above. As alsodescribed above, the planar regular image shown in FIG. 17(B) candisplay a regular person therein for a wide angle, and thus can be usedwith the conventional object recognizing technique.

The monitoring camera system can automatically detect a position of anobject to be monitored and can continuously monitor the detected objectto be monitored by applying the object recognizing technique to theplanar regular image.

Specifically, as shown in FIG. 17(C), a position to be monitored on thefisheye-distorted image corresponding to the position to be monitored,which is automatically detected on the planar regular image by theapplication of the present invention, can be successively (continuously)calculated. The object to be monitored can be automatically traced bymaking the position to be monitored on the fisheye-distorted image asthe center of the cut-out corrected image (that is, by thereconversion).

With a specific explanation of FIG. 17(C), the upper diagrams of FIG.17(C)-(1) to (4) are converted planar regular images (FIG. 17(C) showshow a person is moving from left to right in the order of (1) to (4)).

A position corresponding to the fisheye-distorted image (center diagram)at the detected position on the planar regular image is calculated bythe reconversion. The lower diagrams show the cut-out corrected images(that is, reconverted planar regular images) about the calculatedposition to be monitored on the fisheye-distorted image. The position ofthe object to be monitored, which is automatically detected by theobject recognizing technique, keeps being updated to the center positionof the cut-out corrected image (that is, the arbitrary point Ci(ui, vi))and the reconverted image is displayed with reference to the centerposition, thereby automatically tracing the object to be monitored.

[6. Exemplary Application to TV Conference System]

An example in which the image processing apparatus SS is applied to amonitoring camera system will be described below with reference to FIGS.18 and 19.

FIG. 18 is conceptual diagrams showing an example in which the imageprocessing apparatus SS is applied to a TV conference system when thefisheye lens camera is arranged sideways, and FIG. 19 is conceptualdiagrams showing an example in which the image processing apparatus SSis applied to the TV conference system when the fisheye lens camera isfaced upward.

The image processing apparatus SS according to the present embodimentfurther includes a digital camera (any device capable of outputting afisheye-distorted image to the image processing apparatus SS may beemployed) mounting a fisheye lens (a wide-angle lens or omnidirectionalmirror may be employed) thereon and a display device, and an objectrecognizing unit in addition to the structure of FIG. 6. The objectrecognizing unit may be realized by the CPU and the like.

FIG. 18(A) is a schematic diagram showing an example in which thefisheye lens camera facing in a lateral direction is arranged in aconference room. In this case, for example, the fisheye lens camera isarranged on the wall, and is arranged opposite to a subject person.

FIG. 18(B) shows a planar regular image converted in the present system.As shown in FIG. 18(B), since the planar regular image at a wide angleof about 180 degrees can be obtained, a plurality of conferenceparticipants can be displayed by one fisheye lens camera.

FIG. 18(C) shows an example in which the object recognizing technique isapplied to recognize conference participants (persons) and theconference participants are reconverted with preset points of therecognized persons (such as centers of gravity of the persons) as thearbitrary points Ci (ui, vi) to display the planar regular image.

The conference participants are displayed in split windows and thus canbe observed in detail as the planar regular images at the same time.

As shown in FIG. 18(D), the magnification for enlarging/reducing theimage in size in each window is individually changed, thereby displayingthe persons at a unified size irrespective of the distance to thecamera.

When the fisheye lens camera is faced in a lateral direction, thefisheye-distortion correction is made in the direction of FIG. 5(A).

FIG. 19(A) is a schematic diagram showing an example in which thefisheye lens camera facing upward is arranged in a conference room. Inthis case, for example, the fisheye lens camera is faced upward on thetable and is arranged in parallel to a subject person.

FIG. 19(B) shows a planar regular image converted in the present system.As shown in FIG. 19(B), since the planar regular image at a wide angleof about 360 degrees can be obtained, all the conference participantscan be displayed by one fisheye lens camera.

FIG. 19(C) shows an example in which the object recognizing technique isapplied to recognize the conference participants (persons) and theconference participants are reconverted with preset points of therecognized persons (such as centers of gravity of the persons) as thearbitrary points Ci(ui, vi) to display the planar regular image.

The conference participants are displayed in split windows and thus canbe observed in detail as the planar regular images at the same time.

[7. Exemplary Application to Pan/Tilt/Zoom Camera]

An example in which the image processing apparatus SS is applied to apan/tilt/zoom camera will be described below with a comparison betweenFIGS. 20 and 21.

FIG. 20 is a block diagram showing the case in which the imageprocessing apparatus SS is applied to a pan/tilt/zoom camera, and FIG.21 is conceptual diagrams showing the case in which the image processingapparatus SS is applied to the pan/tilt/zoom camera.

As shown in FIG. 20, in the block diagram showing the case in which theimage processing apparatus SS is applied to a pan/tilt/zoom camera, thepreviously-shown block diagram of the image processing apparatus SS isadded with a digital camera (fisheye lens camera, any device capable ofoutputting a fisheye-distorted image to the image processing apparatusSS may be employed) mounting a fisheye lens (a wide-angle lens oromnidirectional mirror may be employed) thereon, a pan/tilt/zoom camera100 (a monitoring camera of the present application) and a correspondingcoordinate calculating unit 101 (an exemplary position controlling unitof the present application) for fisheye lens camera and pan/tilt/zoomcamera.

The pan/tilt/zoom camera 100 includes a mechanism part capable ofdriving an orientation of the camera lens in the horizontal directionand in the vertical direction, and an optical zoom mechanism.

The corresponding coordinate calculating unit 101 for fisheye lenscamera and pan/tilt/zoom camera outputs a control signal for controllinga position of the pan/tilt/zoom camera 100 in a predetermined directionto the pan/tilt/camera 100. A driving part of the pan/tilt/zoom camera100 is driven to the position indicated by the control signal inresponse to the control signal.

FIG. 21(A) is a conceptual diagram showing the case in which the imageprocessing apparatus SS is applied to the pan/tilt/zoom camera. As shownin FIG. 21(A), the fisheye lens camera and the pan/tilt/zoom camera areconnected to exchange information each other.

It is generally known that the pan/tilt/zoom camera 100 can photographan image with less distortion (that is, can photograph an image withhigh viewability) but has a narrow angle and a blind angle can occurtherein due to pan/tilt. The present embodiment is configured such thatthe image processing apparatus SS is applied to the pan/tilt/zoom camera100 to grasp an object to be monitored from the image photographed bythe wide-angle fisheye lens and to photograph the object to be monitoredas an image with high viewability by the pan/tilt/zoom camera 100.

In order to realize the operations of the present embodiment, thecorresponding coordinate calculating unit 101 for fisheye lens cameraand pan/tilt/zoom camera transmits position information on the object tobe monitored, which is calculated by the fisheye lens camera, to thepan/tilt/zoom camera 100, and thereby controls the pan/tilt/zoom camera.

The concept of the control will be described with reference to FIGS.21(B) to 21(E).

The left diagram in FIG. 21(B) shows a fisheye-distorted imagephotographed by a fisheye lens camera and the right diagram in FIG.21(B) shows a planar regular image photographed by the pan/tilt/zoomcamera 100. When the photographing centers are compared, it can be seenthat a difference occurs between the centers of the coordinate systemsdepending on the camera-arranged position. The corresponding coordinatecalculating unit 101 for fisheye lens camera and pan/tilt/zoom cameracalculates a difference (which will be called “difference in centerpositions” below) between the photographing center of thefisheye-distorted image photographed by the fisheye lens camera (anexemplary first photographing center) and the photographing center ofthe planar regular image photographed by the pan/tilt/zoom camera 100 (asecond photographing center). The difference in center positions may bepreviously stored.

Since the image angle photographed by the fisheye lens is previouslydetermined, an actual distance between the photographing center and anobject present in the photographed image (which will be called “positioninformation obtained from fisheye lens camera” below) can be grasped.Similarly, since the image angle photographed by the pan/tilt/zoomcamera 100 is also previously determined, an actual distance between thephotographing center and an object present in the photographed image(which will be called “position information by the pan/tilt/zoom camera100” below) can be grasped.

The present embodiment is such that distance information for setting thephotographing center of the pan/tilt/zoom camera 100 is calculated basedon the position information obtained from the fisheye lens camera andthe position information by the pan/tilt/zoom camera 10, and a controlsignal for moving the photographing center of the pan/tilt/zoom camera100 to the position indicated by the distance information is output tothe pan/tilt/zoom camera 100.

When a fisheye-distorted image is converted and a planar regular imageis displayed (FIG. 21(C)) and an object to be monitored in the planarregular image is selected (for example, selected by the pointing deviceor recognized by the object recognizing technique) (FIG. 21(D)), thecorresponding coordinate calculating unit 101 for fisheye lens cameraand pan/tilt/zoom camera calculates a coordinate Si of thefisheye-distorted image corresponding to a predetermined position (suchas center of gravity) of the object to be monitored, which is selectedas the center coordinate Ci for the reconversion.

The corresponding coordinate calculating unit 101 for fisheye lenscamera and pan/tilt/zoom camera calculates the difference in centerposition and the position information obtained from the fisheye lenscamera at Si as the distance information, and transmits the controlsignal to the driving part of the pan/tilt/zoom camera 100 such that theposition information obtained from the fisheye lens camera and Si arethe photographing center of the pan/tilt/zoom cameral 100 based on thecalculated distance information.

The pan/tilt/zoom camera 100 performs pan/tilt/zoom thereby tophotograph an object to be monitored (FIG. 21(E)). The present inventionand the pan/tilt/zoom camera are used together, thereby efficientlycontrolling the pan/tilt/zoom camera.

The operations of the image processing apparatus SS when the imageprocessing apparatus SS is applied to the pan/tilt/zoom camera will bedescribed below with reference to FIG. 22.

FIG. 22 is a flowchart showing the operations of the image processingapparatus SS when the image processing apparatus SS is applied to thepan/tilt/zoom camera.

At first, the corresponding coordinate calculating unit 101 for fisheyelens camera and pan/tilt/zoom camera calculates a difference between thephotographing center of the fisheye-distorted image photographed by thefisheye lens camera and the photographing center of the planar regularimage photographed by the pan/tilt/zoom camera 100 (“difference incenter positions”) (step S11).

In order to convert the fisheye-distorted image S and display the planarregular image on the displaying part, when the cut-out center P(X0, Y0)of the fisheye-distorted image S is selected based on the user'soperation of the parameter inputting unit 3 or by using the valuespreset in the image processing apparatus SS (step S12), the selectedinformation is output as the parameter to the correction parametercalculating part 21.

When being given the eye vector n and the planar tilt angle Φ from theparameter inputting unit 3, the correction parameter calculating part 21calculates the azimuth angle α and the zenith angle β based on the eyevector n, and calculates the rotation coefficients A, B, C, D, E, Fbased on Expression (3) to (8) (step S13) and outputs them to thefisheye-distortion correcting unit 1.

The fisheye-distortion correcting unit 1 uses the calculation results(α, β, Φ) input from the correction parameter calculating part 21 toconvert the fisheye-distorted image S into the planar regular image(step S14), and the planar regular image memory controller 52 displaysthe planar regular image on the display screen (step S15).

When an object to be monitored in the planar regular image is selected(step S16: YES), the coordinate Si of the fisheye-distorted imagecorresponding to a predetermined position (such as center of gravity) ofthe object to be monitored, which is selected as the center coordinateCi for the reconversion, is calculated (step S17).

The corresponding coordinate calculating unit 101 for fisheye lenscamera and pan/tilt/zoom camera transmits the control signal to thedriving part of the pan/tilt/zoom camera 100 such that the positioninformation obtained from the fisheye lens camera and the Si are thephotographing center of the pan/tilt/zoom camera 100 based on theposition information obtained from the fisheye lens camera at Si (stepS18).

Then, the pan/tilt/zoom camera 100 performs pan/tilt/zoom thereby tophotograph an object to be monitored (step S19).

The embodiment described above does not limit the invention within thescope of claims. All the combinations of structures described in theembodiment are not necessarily essential for solving the problems of theinvention.

As described above, when receiving an instruction of selecting anarbitrary point Ci(ui, vi) of a 2D-viewable planar regular imagegenerated based on a fisheye-distorted image S photographed by a fisheyelens, the embodiment calculates point Si(xi, yi) on thefisheye-distorted image S as a corresponding coordinate of Ci(ui, vi),and newly generates a pixel information group configuring the planarregular image about Ci(ui, vi) based on a pixel information groupconfiguring the fisheye-distorted image S, and thus can accurately andeffectively recognize an object based on the fisheye-distorted imagephotographed at a wide angle.

There has been described in the embodiment with the case in which thepresent application is applied to the image processing apparatus, butthe embodiment may be applied to electronic devices such as digitalvideo camera, digital camera, personal computer and home appliance.

DESCRIPTION OF REFERENCE NUMERALS

-   1: Fisheye-distortion correcting unit-   2: Correction parameter calculating unit-   3: Parameter inputting unit-   4: Corresponding coordinate calculating unit-   5: Planar regular image creating unit-   6: Fisheye-distorted image memory-   7: Planar regular image memory-   S: Image processing apparatus

1. An image processing apparatus comprising: distorted circular imagestoring unit that stores a pixel information group configuring adistorted circular image generated based on the externally-inputdistorted circular image photographed by a wide-angle lens oromnidirectional mirror; converting unit that converts the stored pixelinformation group configuring the distorted circular image into a pixelinformation group configuring a 2D-viewable planar regular image; planarregular image storing unit that stores the pixel information groupconfiguring the converted planar regular image; selection-instructionreceiving unit that receives an instruction of selecting pixelinformation corresponding to an arbitrary point on the stored planarregular image; corresponding coordinate specifying unit that specifiespixel information on a distorted circular image corresponding to thepixel information on the selection-instructed planar regular image; andreconverting unit that reconverts the stored pixel information groupconfiguring the distorted circular image into a pixel information groupconfiguring a 2D-viewable planar regular image about the specified pixelinformation.
 2. The image processing apparatus according to claim 1,comprising: screen information generating unit that generates screeninformation having a predetermined size to be displayed on a displayingpart based on the pixel information group configuring the reconvertedplanar regular image.
 3. The image processing apparatus according toclaim 1, wherein the selection-instruction receiving unit receives aninstruction of selecting pixel information corresponding to multiplepoints on the planar regular image converted by the converting unit; thecorresponding coordinate specifying unit specifies pixel information ona distorted circular image corresponding to the respective items ofpixel information on the selection-instructed planar regular image, andthe reconverting unit reconverts the stored pixel information groupconfiguring the distorted circular image into a pixel information groupconfiguring a 2D-viewable planar regular image about the respectiveitems of specified pixel information per pixel information specified bythe corresponding coordinate specifying unit.
 4. The image processingapparatus according to claim 3, comprising: display controlling unitthat controls a display form such that each item of the generated screeninformation is displayed at the same size on the displaying part.
 5. Theimage processing apparatus according to any one of claim 1, comprising:object recognizing unit that recognizes a predetermined object displayedon the converted planar regular image, wherein the selection-instructionreceiving unit receives an instruction of selecting pixel informationcorresponding to an arbitrary point on the recognized object.
 6. Animage processing apparatus comprising: distorted circular image storingunit for storing a pixel information group configuring multipledistorted circular images generated based on the externally-inputdistorted circular images obtained by continuously photographing asubject by a wide-angle lens or omnidirectional mirror in correspondenceto each distorted circular image; converting unit that converts thestored pixel information group configuring the distorted circular imageinto a pixel information group configuring a 2D-viewable planar regularimage per distorted circular image; planar regular image storing unitthat stores the converted pixel information group configuring the planarregular image; object recognizing unit that recognizes a specific objectdisplayed on the converted planar regular image; operation recognizingunit that recognizes a successive change indicated by the recognizedobject on the planar regular images per planar regular image; secondcorresponding coordinate specifying unit that specifies pixelinformation on a distorted circular image corresponding to an arbitrarypoint on an object per planar regular image recognized by the operationrecognizing unit; and reconverting unit that reconverts the pixelinformation group configuring the distorted circular image into a pixelinformation group configuring a 2D-viewable planar regular image aboutthe pixel information on each distorted image specified by the secondcorresponding coordinate specifying unit per pixel information specifiedby the second corresponding coordinate specifying unit.
 7. A conferencesystem comprising: a camera for outputting a subject photographed by awide-angle lens or omnidirectional mirror as a distorted circular imageto an image processing apparatus; and the image processing apparatus forgenerating screen information based on the input distorted circularimage and outputting it to a terminal device, the image processingapparatus comprising: distorted circular image storing unit that storesa pixel information group configuring the distorted circular imagegenerated based on the input distorted circular image; converting unitthat converts the stored pixel information group configuring thedistorted circular image into a pixel information group configuring a2D-viewable planar regular image; planar regular image storing unit thatstores the pixel information group configuring the converted planarregular image; object recognizing unit that recognizes one or multiplespecific objects displayed on the converted planar regular image;selection-instruction receiving unit that receives an instruction ofselecting pixel information corresponding to an arbitrary point on therecognized object per recognized object; corresponding coordinatespecifying unit that specifies pixel information on a distorted circularimage corresponding to the pixel information on the selection-instructedplanar regular image per recognized object; reconverting unit thatreconverts the stored pixel information group configuring the distortedcircular image into a pixel information group configuring a 2D-viewableplanar regular image about the pixel information specified by thecorresponding coordinate specifying unit per recognized object; screeninformation generating unit that generates the pixel information groupconfiguring the reconverted planar regular image as screen informationhaving a predetermined size to be displayed on a displaying part perrecognized object; and outputting unit that outputs the generated screeninformation to the terminal device.
 8. The conference system accordingto claim 7, wherein the camera is arranged such that a wide-angle lensor omnidirectional mirror mounted on the camera is faced upward.
 9. Amonitoring system comprising: a monitoring camera mounting thereon afunction of outputting a subject photographed by a wide-angle lens oromnidirectional mirror within a predetermined photographing range from afirst photographing center as a distorted circular image to an imageprocessing apparatus and a function of outputting a subject photographedby a lens group comprising a zoom mechanism within a predetermined rangefrom a second photographing center as a planar regular image to theimage processing apparatus; and the image processing apparatus forgenerating screen information based on the input distorted circularimage, wherein the monitoring camera comprises: pan/tilt driving unitthat switches a photographing position of the camera, and the imageprocessing apparatus comprises: distorted circular image storing unitthat stores a pixel information group configuring the distorted circularimage generated based on the input distorted circular image; convertingunit that converts the stored pixel information group configuring thedistorted circular image into a pixel information group configuring a2D-viewable planar regular image; planar regular image storing unit thatstores the pixel information group configuring the converted planarregular image; object recognizing unit that recognizes an objectdisplayed on the converted planar regular image; selection-instructionreceiving unit that receives an instruction of selecting pixelinformation corresponding to an arbitrary point on the stored planarregular image; corresponding coordinate specifying unit that specifyingpixel information on a distorted circular image corresponding to thepixel information on the selection-instructed planar regular image; andposition controlling unit that calculates distance information for newlysetting the second photographing center at a position indicated by thepixel information specified by the corresponding coordinate specifyingunit based on the previously-calculated first photographing center andsecond photographing center, and outputting a control signal for drivinga monitoring camera to a position indicated by the distance informationto the pan/tilt driving unit.
 10. An image processing method in an imageprocessing apparatus, comprising: a distorted circular image storingstep of storing a pixel information group configuring a distortedcircular image generated based on the externally-input distortedcircular image photographed by a wide-angle lens or omnidirectionalmirror; a converting step of converting the stored pixel informationgroup configuring the distorted circular image into a pixel informationgroup configuring a 2D-viewable planar regular image; a planar regularimage storing step of storing the pixel information group configuringthe converted planar regular image; a selection-instruction receivingstep of receiving an instruction of selecting pixel informationcorresponding to an arbitrary point on the stored planar regular image;a corresponding coordinate specifying step of specifying pixelinformation on a distorted circular image corresponding to the pixelinformation on the selection-instructed planar regular image; and areconverting step of reconverting the stored pixel information groupconfiguring the distorted circular image into a pixel information groupconfiguring a 2D-viewable planar regular image about the pixelinformation specified by the corresponding coordinate specifying step.11. A non-transitory computer-readable storage medium that stores acomputer-image processing program, the image processing program causinga computer to execute: a distorted circular image storing step ofstoring a pixel information group configuring a distorted circular imagegenerated based on the externally-input distorted circular imagephotographed by a wide-angle lens or omnidirectional mirror; aconverting step of converting the stored pixel information groupconfiguring the distorted circular image into a pixel information groupconfiguring a 2D-viewable planar regular image; a planar regular imagestoring step of storing the pixel information group configuring theconverted planar regular image; a selection-instruction receiving stepof receiving an instruction of selecting pixel information correspondingto an arbitrary point on the stored planar regular image; acorresponding coordinate specifying step of specifying pixel informationon a distorted circular image corresponding to the pixel information onthe selection-instructed planar regular image; and a reconverting stepof reconverting the stored pixel information group configuring thedistorted circular image into a pixel information group configuring a2D-viewable planar regular image about the pixel information specifiedby the corresponding coordinate specifying step.